Progress Report on “ Big Data Mining ”

Big Data consists of voluminous, high-velocity and high-variety datasets that are increasingly difficult to process using traditional methods. Data Mining is the process of discovering knowledge by analysing raw datasets. Traditional Data Mining tools, such as Weka and R, have been designed for single-node sequential execution and fail to cope with modern Big Data volumes. In contrast, distributed computing frameworks such as Hadoop and Spark, can scale to thousands of nodes and process large datasets efficiently, but lack robust Data Mining libraries. This project aims to combine the extensive libraries of Weka with the power of the distributed computing frameworks Hadoop and Spark. The system aims to achieve scalability to large volumes by partitioning big datasets and executing Weka algorithms against partitions in parallel. Both frameworks support the MapReduce paradigm. In MapReduce, Map functions process dataset partitions in parallel and Reduce functions aggregate the results. Weka learning algorithms can be enclosed in classes (wrappers) that implement the Map interface and generate models on dataset partitions in parallel. Weka Meta-Learners can be enclosed in the Reduce interface and aggregate these models to a single output. Weka wrappers for the first version of Hadoop, that already exist in Weka packages, were edited and compiled against the second (latest) version. A Hadoop2 cluster was built locally for testing and the system was tested in a variety of classification tasks. The system was then installed on AWS to carry out experiments at larger scales. Preliminary results demonstrate linear scalability. The Spark framework was installed locally and was tested for interoperability with Hadoop MapReduce tasks. As expected since both systems are Java-based, Hadoop tasks can be executed on both systems and the existing solution is possible to be used in Spark. The final part of the project will use this observation and implement wrappers for Weka algorithms on Spark. By taking advantage of its main-memory caching mechanisms, it is possible to greatly improve system performance.